feat: bot detection logic by tkotthakota-adobe · Pull Request #170 · adobe/spacecat-task-processor

tkotthakota-adobe · 2025-12-23T02:33:50Z

Implemented comprehensive bot protection detection and alerting in the Task Processor to identify when sites are blocked by bot protection services (Cloudflare, Akamai, Imperva, etc.) and prevent unnecessary processing.

Using scrape client, retrieves scrape results along with abort info to check on bot protection signal.
If there is a bot protection signals user via slack channel.
Complete flow details are here: https://wiki.corp.adobe.com/display/AEMSites/Spacecat+BOT+detection
https://jira.corp.adobe.com/browse/SITES-37727

Tests:
DynamoDB records created and fetched using scrape client:
Sample record

{
  "pk": "$spacecat#scrapejobid_ef09f4e4-c9cb-4a5f-b884-ac9b3ffa69a3",
  "sk": "$scrapejob_1",
  "abortInfo": {
    "reason": "bot-protection",
    "details": {
      "totalUrlsCount": 1,
      "blockedUrlsCount": 150,
      "blockedUrlsSampled": true,
      "blockedUrls": [
        {
          "url": "https://www.abbvie.com/science/areas-of-innovation/ai-and-data-convergence.html",
          "blockerType": "cloudflare",
          "httpStatus": 403,
          "confidence": 0.99
        }
      ],
      "byBlockerType": {
        "cloudflare": 150
      },
      "byHttpStatus": {
        "403": 150
      },
      "auditType": "meta-tags"
    }
  }
}

Debug Log:

"[BOT-BLOCKED] Bot protection detected"

Tests:
https://cq-dev.slack.com/archives/C060T2PPF8V/p1770848981948789?thread_ts=1770846988.025919&cid=C060T2PPF8V

github-actions · 2025-12-23T03:06:27Z

This PR will trigger no release when merged.

codecov · 2025-12-23T03:31:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…SITES-37727

tkotthakota-adobe · 2026-02-07T19:04:52Z

@solaris007 Appreciate checking the IPs and adding both variables to secrets manager on all 3 environments. Review comments are addressed.

Do we use stage environment in some particular cases which I am not aware of?
Adjusted tests to use right module missed this after moving bot code to bot-detection.js.
IPs were suggested by cursor. I did not verify my bad. Looks like they were obtained from tests not from terraform. I also verified using terraform command. Agree with the IPs you posted. Sorry for the confusion.
log.debug is causing issue in the shared lib. Fixed it to use log.info.

tkotthakota-adobe · 2026-02-08T01:15:31Z

@solaris007 what could be causing this deploy error? Changes made should not effect deployment in my view.
Other services are deployed fine.
https://github.com/adobe/spacecat-task-processor/actions/runs/21789518189/job/62866745175?pr=170

tkotthakota-adobe · 2026-02-08T04:21:11Z

@solaris007 I noticed on aws console lambda is being deployed. For some reason Jenkins deployer is showing the error I shared earlier.

solaris007

Re-review - All requested changes addressed

The 4 issues from the previous review have all been fixed:

Mock paths now correctly target bot-detection.js (verified all esmock calls)
Dead re-export removed - cloudwatch-utils.js only exports getAuditStatus
sortJobsByDate function vs sortedJobs result - no more name collision
Single bot-detection.test.js, no duplicate test file

Bot detection end-to-end flow is clean: handler gets jobId from scrape jobs, checkAndAlertBotProtection reads abortInfo via ScrapeClient, converts to stats, sends Slack alert. LGTM.

Minor cleanup items

Duplicate JSDoc on filterJobsByTimestamp - Two JSDoc blocks stacked on the same function. Remove the shorter one.
Unnecessary esmock mock - bot-detection.test.js mocks fetchRecentThreadMessages from slack-utils.js, but bot-detection.js doesn't import it. Harmless but confusing.
Gist tarball dependencies - 4 packages on gist tarballs. Must be replaced with published npm versions once #1308 merges.
getScrapeJobsByBaseURL no longer filters by 'default' processing type - Intentional? This now returns jobs of all processing types. Just confirming this is desired behavior.

tkotthakota-adobe · 2026-02-09T01:38:52Z

@solaris007 addressed your comments. For some reason this PR is not getting deployed. When I created fresh test PR it deployed fine. When I created new PR out of this PR it failed. That means some state in this PR is causing issue. I am running out of ideas. One other option is just cherry pick code to new PR. This issue started happening only from yesterday.

Update: Upgrading to latest @adobe/helix-deploy (same as in audit worker) solved the issue.

# [1.9.0](v1.8.2...v1.9.0) (2026-02-13) ### Features * bot detection logic ([#170](#170)) ([1741930](1741930))

solaris007 · 2026-02-13T01:41:03Z

🎉 This PR is included in version 1.9.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

bot detection logic

2992d0b

tkotthakota-adobe marked this pull request as draft December 23, 2025 02:33

test fix

0212b84

fix

b4a71a3

tkotthakota-adobe added 24 commits December 23, 2025 11:28

fix tests

e05927b

fix intermitent test failures

d1a3b28

merge main

d27a645

tests

b1e15ab

Merge branch 'main' of github.com:adobe/spacecat-task-processor into …

f5736e6

…SITES-37727

merge main

161a5f0

update shared lib

7e25c7e

read bot protection from scrape metadata

7c11cd9

test

c05acb5

test

842343d

improve tests

6615a8d

Merge branch 'main' of github.com:adobe/spacecat-task-processor into …

8e04b1a

…SITES-37727

merge main + update lib

6621caa

read content scraper logs when bot protection detected

0a46462

read bot protection flag from scrape results

3a80e36

test coverage

f378b77

empty scrape.json to check bot protection

d7f7680

merge main

b3f3681

updated lib

c5123d6

simplify logic

c4f2d82

simplify logic to just read logs for bot protection

8040961

merge main

7054885

adjust logs

4d02b7f

refactor to simplify logic

262dfb1

address review comments

ebd7544

tkotthakota-adobe requested a review from solaris007 February 7, 2026 19:21

tkotthakota-adobe added 4 commits February 7, 2026 16:55

update lib and log fix

b0a28a3

merge main

df238d8

slack lib

21e932b

test fix

9f3e153

solaris007 approved these changes Feb 8, 2026

View reviewed changes

tkotthakota-adobe added 4 commits February 8, 2026 11:06

updated lib

2ccb4e8

add logs + updated libs

e6ba882

remove unused s3 lib

6928533

add test

190a6ba

tkotthakota-adobe added 10 commits February 8, 2026 19:47

revert slack client lib to old version

039dd58

lock file

40640ac

restore s3 lib + slack client

6815d07

update helix deploy lib version

0d9d05d

update lib

b663c3f

support for multiple jobIds for siteUrl

c948411

test improvement

99fb900

show stats always irrespective of bot protection

1b343ff

adjust slack messaging

d12aa01

new tests

ce66e57

tkotthakota-adobe merged commit 1741930 into main Feb 13, 2026
28 checks passed

tkotthakota-adobe deleted the SITES-37727 branch February 13, 2026 01:38

solaris007 pushed a commit that referenced this pull request Feb 13, 2026

chore(release): 1.9.0 [skip ci]

fbc63b0

# [1.9.0](v1.8.2...v1.9.0) (2026-02-13) ### Features * bot detection logic ([#170](#170)) ([1741930](1741930))

solaris007 added the released label Feb 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

feat: bot detection logic#170

feat: bot detection logic#170
tkotthakota-adobe merged 87 commits intomainfrom
SITES-37727

tkotthakota-adobe commented Dec 23, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 23, 2025

Uh oh!

codecov bot commented Dec 23, 2025 •

edited

Loading

Uh oh!

tkotthakota-adobe commented Feb 7, 2026 •

edited

Loading

Uh oh!

tkotthakota-adobe commented Feb 8, 2026

Uh oh!

tkotthakota-adobe commented Feb 8, 2026

Uh oh!

solaris007 left a comment

Uh oh!

tkotthakota-adobe commented Feb 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

solaris007 commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

tkotthakota-adobe commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 23, 2025

Uh oh!

codecov bot commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tkotthakota-adobe commented Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkotthakota-adobe commented Feb 8, 2026

Uh oh!

tkotthakota-adobe commented Feb 8, 2026

Uh oh!

solaris007 left a comment

Choose a reason for hiding this comment

Re-review - All requested changes addressed

Minor cleanup items

Uh oh!

tkotthakota-adobe commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

solaris007 commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tkotthakota-adobe commented Dec 23, 2025 •

edited

Loading

codecov bot commented Dec 23, 2025 •

edited

Loading

tkotthakota-adobe commented Feb 7, 2026 •

edited

Loading

tkotthakota-adobe commented Feb 9, 2026 •

edited

Loading